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ABSTRACT 



This paper discusses the use of standardized testing in a 
university-based executive training program in Japan. The program is an 
English language, cross-cultural, and business training program offered to 
Japanese and Chinese businesspersons preparing to enter the international 
business world. Reports on students, sent to their companies, summarize 
program activities, student progress, student skills and participation level, 
and standardized English test scores. A study was undertaken to assess: (1) 

how, if at all, changes in test scores reflected changes in student 
capabilities, (2) program stakeholders’ perceptions of the practical value of 
the scores, and (3) dilemmas posed by use of the scores in student and 
program reporting and program marketing. Data were gathered from programs 
from 1992 through 1997 and analyzed in this context. It is concluded that 
while the program cannot abandon use of the test and its scores, it also 
cannot continue to use test data exclusively to promote the program. 
Additional information is being sought from sponsoring companies concerning 
in-house skill measurement and training, employee skill improvements they 
expect from program participation, information they would like included in 
student reports, and importance the company attaches to use of the test and 
its scores. (MSE) 
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Abstract 

The presenters will discuss the challenge of balancing qualitative assessment with 
standardized scores in an executive training program in Japan. They will examine 
TOEIC scores from their training programs, comparing them with other assessment 
means, and suggest ways to deal with companies’ emphasis on TOEIC gains. 




Introduction 



The Intensive International Executive Program, or IIEP, is an English, cross-cultural, 
and business training program held four times each year at the International Uni- 
versity of Japan (IUJ), in rural Niigata Prefecture. Participants in the IIEP are 
Japanese and Chinese business people who are preparing to enter the international 
business world, usually by taking a bilingual post in Japan or China or by going 
overseas. 




At the end of each program the IIEP instructors prepare a Report to Company, 
which is sent to the company personnel director or the participants’ division chiefs. 
Each report summarizes the major activities of each program, describes the progress 
made by the participant, comments on the adaptability, cross-cultural skills, lead- 
ership, and participation levels of the participant, and reports on the participants’ 
scores on the Test of English for International Communication, or TOEIC, which is 
administered twice during each program. 

Over the years it has become clear that TOEIC is the only English language assess- 
ment that is familiar to all the stakeholders in the IIEP, defined as the program par- 
ticipants, program instructors, and administrative officers and staff related to both, 
i.e., at IUJ and at the sponsoring companies. Programs that achieve a substantial 
average TOEIC increase are deemed successes by many program stakeholders, and 
the effectiveness of programs that did not end in TOEIC gains for their employees 
has been questioned by these participants and their sponsoring companies. 

Recognizing that TOEIC cannot be eliminated from the program, the IIEP has 
developed several additional methods of measuring and reporting progress, partic- 
ipation, leadership, and other performance and skills categories, but the primary 
administrative stakeholders in the program and many participants continue to focus 
on TOEIC. In an attempt to accurately convey to companies what they can expect 
from individual IIEPs (of four-, eight-, and 10- week duration) in terms of TOEIC 
gains, the IIEP instructors examined the results from 400 TOEICs from the period 
1992 - 1997. 

Parts of this analysis are reported below. To summarize, the analysis found no 
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pattern of TOEIC improvement that could be used to help companies match partic- 
ipants to individual programs or to predict the success of their participants in any 
single program. The corollary of this is of course that the IIEP is unable to predict 
individual or program gains based on program-specific average TOEIC increases 
from past programs. 

Background on the IIEP and the use of TOEIC 

In 1990 and 1991 the IIEP ran as a morning course in IUJ’s Intensive English Pro- 
gram (IEP). The IEP prepares entering IUJ students for their two-year course of 
study in either international management or international relations. IIEP partici- 
pants met for three hours Monday through Friday in a course entitled “Text Skills 
IIEP.” This course dealt with business case studies, cross-cultural readings and ac- 
tivities, and leading meetings and discussions. 

In the afternoons the IIEP participants were integrated with the IEP communication 
skills classes, which focused almost exclusively on presentation skills. The IIEP 
participants also took part in all IEP curricular and social activities, sis well as the 
IEP pre- and post-program testing. This testing included TOEFL, and from 1990 
through 1993 all IIEP participants took the TOEFL twice - at the beginning and 
end of the program. 

In 1992 the IIEP ran separate from the IEP except for major social activities and 
computer instruction, which combined IIEP and IEP participants. At the request 
of company sponsors and at the suggestion of key IUJ administrators, the IIEP 
introduced TOEIC during the 1992 program as an optional addition to the required 
TOEFL exam. In 1992 half the participants chose to take both tests. In 1993 all 
but one participant took both. From 1993 the TOEFL was dropped, and TOEIC 
took its place. 

The Current Program 

In January 1996 the IIEP began to run throughout the year, as an eight-week Winter 
Program, a ten- week Summer Program, and four- week Fall and Spring Programs. 
All four programs follow the same basic daily schedule, divided into morning classes 
devoted to language instruction, and afternoon classes devoted to business training 
and cross-cultural activities. 

The Morning Class curriculum is divided into two 90-minute classes that meet five 
days a week and total 15 hours of weekly instruction. The first class, International 
Business English, is built around a Business English text and listening and speaking 
skills are emphasized. The second class, Communication Skills for International 
Managers, consists mainly of participants leading Business Meetings and delivering 
Oral Presentations. The underlying philosophy of the Morning Class is to provide 
a familiar teacher-led environment during the first class and then move to a more 
participant-led focus in the second. 

Business Writing plays a secondary role in the Morning Classes. Participants write 
weekly Business Journals on specific business topics, which are commented upon 
and corrected for both content and grammar. Other assessment in the Morning 



Classes includes weekly comprehension quizzes and videotaped Oral Presentations 
and Business Meetings. Comments from the teacher about participants on Business 
Journals, quizzes, presentations, and meetings are given in the final Reports to 
Company. 

Afternoon sessions offer a wide variety of activities organized by IUJ’s MBA and in- 
ternational relations faculty, including cross-cultural activities and interaction with 
IUJ’s international student body and frequent business presentations given by par- 
ticipants based on Internet research into business and culture in Asia. Afternoon 
assessments evaluate participation levels, oral presentations, case discussion skills, 
cross-cultural management skills, and leadership skills based on observations made 
by the IIEP and IUJ MBA and international relations faculty. Outstanding achieve- 
ments by each participant on any afternoon activity are described in the Report to 
Company. 

IIEP Stakeholders and TOEIC 

IIEP stakeholders include all those at IUJ and at the sponsoring companies who 
have a stake, or interest, in ensuring that the program is effective. Table 1 shows 
the attitudes toward TOEIC held by the major IIEP stakeholders. 



Table 1: IIEP Stakeholders and their Attitudes toward TOEIC 



Program Stakeholders 


General Stakeholder Attitudes toward TOEIC 


program participants 


Varied. About one half report that TOEIC scores 
are required for promotion or for postings overseas 


personnel staff of 
sponsoring companies 


Virtually all report that TOEIC is used in 
the company, along with “Eiken” rankings. 


division or section chiefs 
at sponsoring companies 


Anecdotal evidence is clear: Middle-level management expect 
participants to return to the office with TOEIC increases. 


IIEP visiting instructors 


Generally negative to neutral about TOEIC, 
particularly about TOEIC preparatory sessions. 


IIEP staff 


Recognize the importance of the TOEIC, and 
assist in the administration of the test. 


IUJ management from 
the business world 


Stress the importance of TOEIC to potential sponsors, but 
recognize that communicative skills are more important. 


IIEP and IUJ marketing 
staff in Tokyo and Niigata 


Place great importance on TOEIC gains, and include 
mention of program gains in all IIEP ad materials. 


IUJ MBA and International 
Relations content faculty 
who contribute to the IIEP 


Increasingly aware of the language 
learning goals of the program, but 
TOEIC is far from their primary concerns. 



Problems with the TOEIC: Reporting and Marketing Dilemmas 

The presence of TOEIC in the IIEP creates the expectation from most stakeholders 
that TOEIC scores will increase during the program. While 80% of all IIEP TOEIC 
scores have increased, these gains have not been consistent or predictable from one 
program to the next, causing some corporate stakeholders to question the quality of 
individual IIEPs and raising the question of what companies can expect from each 

program in terms of TOEIC. 
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While maintaining the TOEIC in the program, the IIEP instructors have sought 
ways to influence the importance corporate sponsors place on TOEIC. This can 
be seen in the different ways that TOEIC has been dealt with in the Report to 
Company. 



Table 2: Reporting TOEIC Results to Sponsoring Companies 



Program 


Reporting TOEIC Results 


Summer 92 


No mention of TOEIC scores on the official report. Scores and a brief 
analysis of them given in a personal letter written to each participant. 


Summer 93 


TOEIC scores included at the end of the company report, along with class 
averages for the test. TOEIC is not mentioned in the program summary at 
the beginning of the report. 


Summer 94 


TOEIC included on first page of the report, along with TOEFL scores and 
the results of an in-house oral proficiency exam. No mention of TOEIC 
in the text of the report. 


Summer 95 


TOEIC listed prominently in second section of the report, along with details 
of each participant’s score and their score relative to the group. Attempt 
at explaining TOEIC losses. 


Winter 96 


TOEIC score listed in section 1 of the report. Program vocabulary test scores 
included alongside TOEIC scores. Attempt made to stress that TOEIC is but one 
element of the program’s testing and evaluation. 


Spring 96 


TOEIC mentioned in a single sentence at the end of the report. Only the first, 
second, and gain/loss figures are given. 


Summer 96 


TOEIC included for the first time in the program summary at the beginning of 
the report. TOEIC results moved to the end of the report. 


Fall 96 


TOEIC moves back to the first section of the report, with a disclaimer that 
TOEIC scores were not as encouraging as they had been in the past. 


Winter 97 


TOEIC moves back to the penultimate section of the report. 


Spring 97 


TOEIC on the first page of the report. 


Fall 97 


TOEIC comprises first major section of the report, following the summary, and 
for the first time comparisons are made with previous TOEIC scores achieved by 
participants from the same companies. Increases relative to other participants 
are included when favorable for the participant. Report mentions past average 
TOEIC increases for the initial TOEIC score range of the participant, but only 
when the participant exceeds the average for that range. 



The Scores 

TOEIC is mentioned in all IIEP marketing materials, yet it is not clear from the 
TOEIC scores just who will achieve TOEIC gains in which program. Nor is it clear 
which aspects of any individual program can be offered as an explanation of why 
any single TOEIC score rose or fell during the program. Table 3 shows the size of 
each program, the amount of time devoted to TOEIC prep, score ranges, and the 
percentage of each group that did not show any TOEIC increase. 
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Table 3: TOEIC Related to Program Size, Number of TOEIC Prep Hours, 
Average TOEIC Increase, Range of TOEIC Scores, 

Percentage with No TOEIC Gains, and Range of TOEIC Entry Scores 



Program 


Size 


TOEIC 

Prep 

Hours 


Average 

Increase 


High 

Gain 


Low 

Gain 


Range 

of 

Gain 


% with 
No 
Gain 


High/Low 
Entry TOEIC 
Range 


Summer 92 


28 


0 


19 


140 


-65 


205 


33 


220 


Summer 93 


29 


0 


21 


120 


-110 


230 


40 


385 


Summer 94 


39 


0 


135 


275 


0 


275 


3 


650 


Summer 95 


28 


0 


37 


150 


-55 


205 


24 


405 


Winter 96 


5 


2 


5 


265 


-40 


305 


50 


270 


Spring 96 


6 


2 


101 


115 


90 


25 


0 


160 


Summer 96 


25 


4 


93 


265 


-40 


305 


6 


510 


Fall 96 


7 


4 


1 


55 


-40 


95 


57 


550 


Winter 97 


7 


4 


126 


235 


35 


200 


0 


495 


Spring 97 


11 


6 


27 


115 


-75 


190 


33 


275 


Summer 97 


23 


12 


106 


250 


-30 


280 


5 


500 


Fall 97 


9 


4 


38 


120 


-30 


150 


22 


535 



To summarize, the overall pattern of TOEIC increase is encouraging, but it is diffi- 
cult to capture the TOEIC gain/loss patterns for any individual program, or from 
one program to the next. This is true also for the relationship between entry-level 
TOEIC scores and TOEIC gains. When viewed collectively, the scores show that 
entry TOEIC scores from 200 - 600 have achieved the greatest gains, as might be 
expected. However, this is far too great a range to be of use to companies who have 
to select participants for the program, and in any case the entry/gain argument 
breaks down at the program level, i.e., when entry level scores for each program are 
compared to the individual TOEIC increases for the participants in those programs. 

Reporting TOEIC: Anomalies 

Adding to the overall problem of reporting TOEIC scores to sponsoring companies 
are those scores that seem to be anomalies, as for example when two participants 
from the same company with similar skills achieve very different TOEIC gains, 
when a participant strong in every other area drops in score, or when a participant’s 
listening and reading increases vary to a great degree. Table 4 lists listening and 
reading increases for 19 participants whose listening/reading improvements differed 
by 100 or more points. 
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Table 4: TOEIC Anomalies 



Listening Improvement 


Reading Improvement 


Range 


180 


-40 


220 


110 


5 


115 


105 


-5 


110 


105 


-40 


145 


95 


-40 


130 


80 


-45 


125 


70 


-30 


100 


45 


-120 


165 


10 


110 


120 


5 


125 


130 


0 


120 


120 


-10 


105 


115 


-10 


95 


105 


-15 


125 


140 


-30 


95 


125 


-45 


65 


110 


-55 


45 


100 


-60 


45 


105 


-75 


80 


155 



It always tempting to ascribe drops in reading scores to the participant’s inability 
to finish the TOEIC in time. However, only one listening/reading improvement pair 
(45 /t 120) in Table 4 would warrant this explanation. 

The Future: Alternative Approaches to TOEIC 

The IIEP accepts the challenge of training any participant, regardless of their En- 
glish language skills, and each program is tailor-made for each group, within the 
common morning and afternoon framework discussed above. The program instruc- 
tors begin to make adaptations to each program as soon as participants applications 
begin to arrive, generally 2-6 weeks before each program, and continue to do so 
until the last week of each program. 

Also, the IIEP trains and evaluates each group independently of other groups. Par- 
ticipants enter the IIEP only once, and so cannot be tracked from one program 
to another, or from one level to another. Indeed, the concept of level does not 
fit with the IIEP in general - participants arrive with different skills sets, respond 
to challenges of various types with those skills sets, and are assessed as individuals 
according to how well they were able to apply and expand their skills during the pro- 
gram. Consequently, the Report to Company concentrates on what each participant 
did, how much they participated, to what degree and in what areas they progressed, 
and what kind of training the participant should pursue after the program. 

Given the importance of TOEIC in Japan, its familiarity with the IIEP stakeholders, 
and recent assurances from ETS that the test will be improved in the near future, 
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the IIEP cannot afford to give up the TOEIC. At the same time, as this paper has 
tried to show, the IIEP also cannot sell TOEIC increases to its sponsoring companies 
on a program-to-program basis, despite the fact that 80% of IIEP participants have 
achieved TOEIC gains over the years. 

Rather than attempt to alter the IIEP to achieve more reliable program-specific 
TOEIC gains, and rather than create an in-house assessment system that is unlikely 
to replace TOEIC in the minds of sponsoring companies, the IIEP has begun to 
gather more information from companies: (1) about how sponsoring companies 

measure English improvement in-house, and what kind of training they provide; (2) 
about what improvements they expect their participants to make in the IIEP; (3) 
about what specific information they would like to have included in the Report to 
Company; and (4) about the relationship between each participant and TOEIC, 
i.e., the importance they attach to TOEIC, how many times they have taken it, 
their experience with other standardized tests, what self-study or organized English 
courses they have pursued, what score they hope to attain during the program, and 
to what extent they would like to be a part of TOEIC prep classes during each 
program. 

The goal is to make TOEIC an individual priority, not a program one, and so an 
aspect of each program that we can report to companies in terms of the individual 
participant’s needs and focus. Reaching this goal would bring the TOEIC in line 
with other elements of the IIEP that focus on individual achievement. 
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